<html>
<head><meta charset="utf-8"><title>new aarch64 ci server · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html">new aarch64 ci server</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="242181363"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242181363" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242181363">(Jun 10 2021 at 09:42)</a>:</h4>
<p>ok the new server is online, configuring it</p>



<a name="242182155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242182155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242182155">(Jun 10 2021 at 09:49)</a>:</h4>
<p>closing the tree as I need to switch the two bare metal servers and their github actions runners</p>



<a name="242182323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242182323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242182323">(Jun 10 2021 at 09:51)</a>:</h4>
<p>closed the tree</p>



<a name="242182548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242182548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242182548">(Jun 10 2021 at 09:54)</a>:</h4>
<p>finished shutting down the existing runners</p>



<a name="242182628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242182628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242182628">(Jun 10 2021 at 09:55)</a>:</h4>
<p>deploying the new server configuration</p>



<a name="242183627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242183627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242183627">(Jun 10 2021 at 10:04)</a>:</h4>
<p>ok whelp our setup doesn't like ubuntu 20.04</p>



<a name="242183640"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242183640" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242183640">(Jun 10 2021 at 10:04)</a>:</h4>
<p>restarting the old runners and reopening the tree while I figure this out</p>



<a name="242198879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242198879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242198879">(Jun 10 2021 at 12:38)</a>:</h4>
<p>ok I'm very confused now</p>



<a name="242204746"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242204746" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242204746">(Jun 10 2021 at 13:24)</a>:</h4>
<p>ok so, at least for the packer build, running the VM without KVM (emulated) seems to work but with KVM (virtualized) the VM just doesn't start without any information on it</p>



<a name="242204808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242204808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242204808">(Jun 10 2021 at 13:24)</a>:</h4>
<p>emulation is slooooooooooooow though, let's hope it's only required for building the image and not for running the CI VMs</p>



<a name="242204822"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242204822" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242204822">(Jun 10 2021 at 13:24)</a>:</h4>
<p>(it's not practical to use emulation for CI, at all)</p>



<a name="242206364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242206364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242206364">(Jun 10 2021 at 13:34)</a>:</h4>
<p>ok gah even running the CI VMs with KVM doesn't work</p>



<a name="242206636"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242206636" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242206636">(Jun 10 2021 at 13:36)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="116122">@simulacrum</span> <span class="user-mention" data-user-id="275830">@Robin Randhawa</span></p>



<a name="242206687"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242206687" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242206687">(Jun 10 2021 at 13:36)</a>:</h4>
<p>can you say more about why the setup doesn't work on 20.04?</p>



<a name="242206777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242206777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242206777">(Jun 10 2021 at 13:36)</a>:</h4>
<p>I'm not sure whether it's ubuntu 20.04, my current guess is more about qemu and the new cpu</p>



<a name="242207148"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242207148" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242207148">(Jun 10 2021 at 13:39)</a>:</h4>
<p>if I start the vm with emulation it works fine, if I start it with virtualization (KVM) there is just one core instantly spinning at 100% all the time</p>



<a name="242207206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242207206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242207206">(Jun 10 2021 at 13:39)</a>:</h4>
<p>and I've not been able to get anything out of qemu, probably because it starts looping before the kernel is able to output anything</p>



<a name="242208806"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242208806" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242208806">(Jun 10 2021 at 13:49)</a>:</h4>
<div class="codehilite" data-code-language="Diff"><pre><span></span><code><span class="gd">--- bar 2021-06-10 15:48:54.168791944 +0200</span>
<span class="gi">+++ baz 2021-06-10 15:48:45.268922015 +0200</span>
<span class="gu">@@ -1,12 +1,12 @@</span>
 "-bios" "build/QEMU_EFI.fd"
 "-boot" "c"
<span class="gd">-"-cpu" "host"</span>
<span class="gi">+"-cpu" "cortex-a57"</span>
 "-device" "scsi-hd,bus=scsi0.0,drive=drive0"
 "-device" "virtio-net,netdev=user.0"
 "-device" "virtio-scsi-pci,id=scsi0"
 "-drive" "if=none,file=build/aarch64/rootfs.qcow2,id=drive0,cache=writeback,discard=unmap,format=qcow2"
 "-m" "512M"
<span class="gd">-"-machine" "type=virt,gic_version=3,accel=kvm"</span>
<span class="gi">+"-machine" "type=virt,accel=tcg"</span>
 "-name" "rootfs.qcow2"
 "-netdev" "user,id=user.0,hostfwd=tcp::PORT-:22"
 "-nographic"
</code></pre></div>



<a name="242208873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242208873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242208873">(Jun 10 2021 at 13:49)</a>:</h4>
<p>like, that's the flags diff between running with kvm and running with emulation</p>



<a name="242208988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242208988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242208988">(Jun 10 2021 at 13:50)</a>:</h4>
<p>which is the minimum set of changes required to enable kvm (kvm only works with <code>host</code>, and acceleration <em>doesn't</em> work with <code>host</code>)</p>



<a name="242209062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209062">(Jun 10 2021 at 13:50)</a>:</h4>
<p>hm, yeah, I know ~nothing about qemu so hard to say</p>



<a name="242209154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209154">(Jun 10 2021 at 13:51)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> do you know anyone at arm who might have insights?</p>



<a name="242209188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209188">(Jun 10 2021 at 13:51)</a>:</h4>
<p>I don't have that much more time to dedicate to investigating this</p>



<a name="242209239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209239">(Jun 10 2021 at 13:51)</a>:</h4>
<p>if we can't figure it out we should probably just keep the current server we have, even though it's less powerful</p>



<a name="242209353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209353">(Jun 10 2021 at 13:52)</a>:</h4>
<p>I'm worried that we <em>do</em> need to update to 20.04</p>



<a name="242209368"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209368" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209368">(Jun 10 2021 at 13:52)</a>:</h4>
<p>like, 18.04 is EOL already I think (or will be soon)</p>



<a name="242209460"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209460" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209460">(Jun 10 2021 at 13:53)</a>:</h4>
<p>where does this script live? (so I can play around with it?)</p>



<a name="242209466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209466">(Jun 10 2021 at 13:53)</a>:</h4>
<p>it should go EOL in 2023</p>



<a name="242209556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209556">(Jun 10 2021 at 13:53)</a>:</h4>
<p>so we have time, and well, I think the problem is not about the ubuntu version but about the new cpu</p>



<a name="242209656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209656">(Jun 10 2021 at 13:54)</a>:</h4>
<p><a href="https://github.com/rust-lang/gha-self-hosted/tree/master/images/ubuntu">https://github.com/rust-lang/gha-self-hosted/tree/master/images/ubuntu</a></p>



<a name="242209696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209696">(Jun 10 2021 at 13:54)</a>:</h4>
<div class="codehilite"><pre><span></span><code>make aarch64-host  # kvm
make aarch64-emul # emulated
</code></pre></div>



<a name="242209730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242209730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242209730">(Jun 10 2021 at 13:54)</a>:</h4>
<p>you need an aarch64 host tho to run <code>aarch64-host</code> (the one that gives us trouble)</p>



<a name="242210818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242210818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242210818">(Jun 10 2021 at 14:01)</a>:</h4>
<p>hm, ok</p>



<a name="242210897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242210897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242210897">(Jun 10 2021 at 14:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> what I can do I think is to start a spot <code>c6g.metal</code> aws instance (0.6$/hr) with ubuntu 20.04 and try to run the script there</p>



<a name="242210974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242210974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242210974">(Jun 10 2021 at 14:02)</a>:</h4>
<p>I'm not opposed</p>



<a name="242211049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242211049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242211049">(Jun 10 2021 at 14:02)</a>:</h4>
<p>not sure you need metal -- iirc, kvm works through virtualization too usually</p>



<a name="242211218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242211218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242211218">(Jun 10 2021 at 14:03)</a>:</h4>
<p>iirc aws doesn't allow nested virtualization</p>



<a name="242211395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242211395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242211395">(Jun 10 2021 at 14:04)</a>:</h4>
<p><a href="https://forums.aws.amazon.com/thread.jspa?threadID=329475">https://forums.aws.amazon.com/thread.jspa?threadID=329475</a></p>



<a name="242211777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242211777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242211777">(Jun 10 2021 at 14:06)</a>:</h4>
<p>actually, graviton2 (<code>c6g</code>) also uses neoverse n1 cores, while graviton uses the previous generation</p>



<a name="242211802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242211802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242211802">(Jun 10 2021 at 14:07)</a>:</h4>
<p>if it doesn't work on a <code>c6g</code> I'll also try on an <code>a1.metal</code></p>



<a name="242214022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242214022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242214022">(Jun 10 2021 at 14:20)</a>:</h4>
<p>Hi. Hmm. Not sure what's up there. Help me understand this please. You're trying to run qemu on the (new) Ampere server right ?</p>



<a name="242214151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242214151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242214151">(Jun 10 2021 at 14:21)</a>:</h4>
<p>And then you're trying to get it to emulate an Arm CPU (cortex-a57) by the looks of it.</p>



<a name="242214609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242214609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242214609">(Jun 10 2021 at 14:24)</a>:</h4>
<p>If I could get the breaking use-case nailed for replication at our end that would be tops but I'm confused with the terminology here! Sorry! :(</p>



<a name="242231139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231139">(Jun 10 2021 at 16:09)</a>:</h4>
<p>let me clarify</p>



<a name="242231190"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231190" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231190">(Jun 10 2021 at 16:09)</a>:</h4>
<p>what we want is to simply have a normal virtual machine with KVM and QEMU</p>



<a name="242231319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231319">(Jun 10 2021 at 16:10)</a>:</h4>
<p>without any kind of emulation, both to be as close to the real hardware as possible and because emulation is <em>sloooooooooow</em></p>



<a name="242231339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231339">(Jun 10 2021 at 16:10)</a>:</h4>
<p>but that's not working</p>



<a name="242231457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231457">(Jun 10 2021 at 16:11)</a>:</h4>
<p>to test whether it was a QEMU problem I tried forcing it to emulate an Arm CPU, and there it worked, but it's not what we want to achieve in prod</p>



<a name="242231465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242231465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242231465">(Jun 10 2021 at 16:11)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> is that more clear?</p>



<a name="242234899"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242234899" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242234899">(Jun 10 2021 at 16:36)</a>:</h4>
<p>ok a graviton2 instance on ubuntu 20.04 shows the same bad behavior :(</p>



<a name="242235390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242235390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242235390">(Jun 10 2021 at 16:40)</a>:</h4>
<p>now trying a c6g.metal with ubuntu 18.04</p>



<a name="242237735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242237735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242237735">(Jun 10 2021 at 16:58)</a>:</h4>
<p>ok it seems to exhibit the same behavior on a c6g.metal with ubuntu 18.04 too, I'll let it time out to be extra sure but the symptoms are the same <span class="user-mention" data-user-id="116122">@simulacrum</span></p>



<a name="242237851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242237851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242237851">(Jun 10 2021 at 16:59)</a>:</h4>
<p>that doesn't give me any reason to believe upgrading to ubuntu 20.04 on the old server would cause problems, as I get the same behavior on a neoverse-n1 core both on 18.04 and 20.04</p>



<a name="242238323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242238323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242238323">(Jun 10 2021 at 17:02)</a>:</h4>
<p>yeah it also fails on a c6g.metal with ubuntu 18.04</p>



<a name="242238387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242238387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242238387">(Jun 10 2021 at 17:03)</a>:</h4>
<p>so I think the two next steps if we want to debug this is:</p>
<ul>
<li>try with a newer ubuntu VM image</li>
<li>try with a newer QEMU version</li>
</ul>



<a name="242238423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242238423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242238423">(Jun 10 2021 at 17:03)</a>:</h4>
<p>but at least we know we can safely upgrade to 20.04 on the old <a href="http://packet.net">packet.net</a> server</p>



<a name="242663892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242663892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242663892">(Jun 14 2021 at 21:53)</a>:</h4>
<p><span class="user-mention" data-user-id="117568">@Aidan Hobson Sayers</span> created the <code>ci-arm-new.infra.rust-lang.org</code> DNS record</p>



<a name="242663941"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242663941" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242663941">(Jun 14 2021 at 21:54)</a>:</h4>
<div class="codehilite"><pre><span></span><code>ssh ci-arm-1-new.infra.rust-lang.org -J bastion.infra.rust-lang.org
</code></pre></div>



<a name="242664227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242664227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242664227">(Jun 14 2021 at 21:57)</a>:</h4>
<p>and the script is here: <a href="https://gist.github.com/pietroalbini/24e90b455bd53f5b10ae97448948ad77">https://gist.github.com/pietroalbini/24e90b455bd53f5b10ae97448948ad77</a></p>



<a name="242995598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242995598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242995598">(Jun 17 2021 at 09:58)</a>:</h4>
<p>spawned an a1.metal instance on us-east-1</p>



<a name="242996858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242996858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242996858">(Jun 17 2021 at 10:12)</a>:</h4>
<p>uh ok the a1.metal failed too</p>



<a name="242997425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242997425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242997425">(Jun 17 2021 at 10:19)</a>:</h4>
<p>even trying ubuntu 21.04 yields the same results</p>



<a name="242997455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242997455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242997455">(Jun 17 2021 at 10:19)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> sooo, we're still having trouble making VMs work on the new machine</p>



<a name="242997536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242997536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242997536">(Jun 17 2021 at 10:20)</a>:</h4>
<p><span class="user-mention silent" data-user-id="117568">Aidan Hobson Sayers</span> will look into the problem a bit more in the weekend, but if we don't find a solution we'll probably have to keep the old server running and "return" the new one</p>



<a name="242998484"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998484" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998484">(Jun 17 2021 at 10:31)</a>:</h4>
<p>Hi <span class="user-mention" data-user-id="121055">@Pietro Albini</span>! Apologies: I got massively side tracked.</p>



<a name="242998511"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998511" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998511">(Jun 17 2021 at 10:31)</a>:</h4>
<p>Hmm. All this does sound very cumbersome indeed!</p>



<a name="242998529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998529">(Jun 17 2021 at 10:31)</a>:</h4>
<p>So what I'll do is:</p>



<a name="242998605"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998605" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998605">(Jun 17 2021 at 10:32)</a>:</h4>
<p>I'll position the problem to a bunch of our qemu/KVM/native-aarch64-silicon folks as follows:</p>



<a name="242998695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998695">(Jun 17 2021 at 10:33)</a>:</h4>
<p>Please verify that it is possible to run qemu + KVM "as-is" (without any emulation) on the servers in question.</p>



<a name="242998702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998702">(Jun 17 2021 at 10:33)</a>:</h4>
<p>I'll aim to get back ASAP.</p>



<a name="242998796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998796">(Jun 17 2021 at 10:34)</a>:</h4>
<p>I do have to ask though (and apologies in advance if this is naive): Why must you use qemu + KVM at all when you have the machine ready to run natively ?</p>



<a name="242998921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242998921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242998921">(Jun 17 2021 at 10:35)</a>:</h4>
<p>mostly for two reasons:</p>
<ul>
<li>multiple CI builds can run at the same time without them interfering with each other</li>
<li>each VM can be discarded as soon as the build finishes, providing a clean environment every time</li>
</ul>



<a name="242999037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999037">(Jun 17 2021 at 10:36)</a>:</h4>
<p>the second reason is fairly important to prevent CI issues</p>



<a name="242999079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999079">(Jun 17 2021 at 10:36)</a>:</h4>
<p><span class="user-mention silent" data-user-id="275830">Robin Randhawa</span> <a href="#narrow/stream/242791-t-infra/topic/new.20aarch64.20ci.20server/near/242998695">said</a>:</p>
<blockquote>
<p>Please verify that it is possible to run qemu + KVM "as-is" (without any emulation) on the servers in question.</p>
</blockquote>
<p>for a bit more details, we're using ubuntu (18.04 or 20.04) as the host and ubuntu 20.04 as the guest</p>



<a name="242999154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999154">(Jun 17 2021 at 10:37)</a>:</h4>
<p>and the QEMU invocation is <a href="https://gist.github.com/pietroalbini/24e90b455bd53f5b10ae97448948ad77">https://gist.github.com/pietroalbini/24e90b455bd53f5b10ae97448948ad77</a> (which is the one generated by <a href="http://packer.io">packer.io</a>)</p>



<a name="242999208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999208">(Jun 17 2021 at 10:38)</a>:</h4>
<p>but yeah that sounds good <span class="user-mention" data-user-id="275830">@Robin Randhawa</span>!</p>



<a name="242999328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999328">(Jun 17 2021 at 10:39)</a>:</h4>
<p>Got it! Thanks!</p>



<a name="242999856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/242999856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#242999856">(Jun 17 2021 at 10:45)</a>:</h4>
<p>One thing that just hit me: Could you add '-d in_asm -D /tmp/qemu.log' to the qemu command line switches and check what ends up in qemu.log ?</p>



<a name="243000054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243000054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243000054">(Jun 17 2021 at 10:47)</a>:</h4>
<p>Also: if possible, could you provide me with a temporary access to the machine ?</p>



<a name="243000333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243000333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243000333">(Jun 17 2021 at 10:50)</a>:</h4>
<p>can you DM me your SSH key?</p>



<a name="243000771"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243000771" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243000771">(Jun 17 2021 at 10:55)</a>:</h4>
<p><span class="user-mention silent" data-user-id="275830">Robin Randhawa</span> <a href="#narrow/stream/242791-t-infra/topic/new.20aarch64.20ci.20server/near/242999856">said</a>:</p>
<blockquote>
<p>One thing that just hit me: Could you add '-d in_asm -D /tmp/qemu.log' to the qemu command line switches and check what ends up in qemu.log ?</p>
</blockquote>
<p><code>qemu.log</code> is empty...</p>



<a name="243011960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243011960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243011960">(Jun 17 2021 at 12:50)</a>:</h4>
<p>Hi <span class="user-mention" data-user-id="121055">@Pietro Albini</span>. I think I figured it out.</p>



<a name="243011984"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243011984" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243011984">(Jun 17 2021 at 12:50)</a>:</h4>
<p>Do you have a repo where the run script lies that I can use to generate a patch ?</p>



<a name="243012010"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012010" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012010">(Jun 17 2021 at 12:51)</a>:</h4>
<p>I also want to use the opportunity to clean/simplify the script more generally.</p>



<a name="243012185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012185">(Jun 17 2021 at 12:52)</a>:</h4>
<p>The biggest issue is that the UEFI binary is too old and buggy. The host system's qemu installation on Debian derived distros supplies a verified version in the qemu-efi-aarch64 package.</p>



<a name="243012347"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012347" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012347">(Jun 17 2021 at 12:54)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> thanks for looking into it <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="243012358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012358">(Jun 17 2021 at 12:54)</a>:</h4>
<p>the UEFI bin is downloaded @ <a href="https://github.com/rust-lang/gha-self-hosted/blob/cb8ccffec19792fe84ae3cd04402ae4de0391f26/images/ubuntu/Makefile#L29-L31">https://github.com/rust-lang/gha-self-hosted/blob/cb8ccffec19792fe84ae3cd04402ae4de0391f26/images/ubuntu/Makefile#L29-L31</a></p>



<a name="243012395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012395">(Jun 17 2021 at 12:55)</a>:</h4>
<p>and it's passed to QEMU @ <a href="https://github.com/rust-lang/gha-self-hosted/blob/cb8ccffec19792fe84ae3cd04402ae4de0391f26/images/ubuntu/ubuntu.json#L108">https://github.com/rust-lang/gha-self-hosted/blob/cb8ccffec19792fe84ae3cd04402ae4de0391f26/images/ubuntu/ubuntu.json#L108</a></p>



<a name="243012409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012409">(Jun 17 2021 at 12:55)</a>:</h4>
<p>Also, can I ask the following:</p>
<ol>
<li>You need to run the qemu instance headless right ? (I ask because the script asks qemu to use a VirtIO GPU but that shouldn't be needed for a headless system).</li>
<li>Do you care about having access to the qemu instance's monitor interface ? (I ask because at present the script does end up getting access to the monitor but that seems unnecessary)</li>
<li>Do you care about logging the kernel's output ? (I ask because at present, qemu is told to effectively discard the console output from the kernel session)</li>
</ol>



<a name="243012868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012868">(Jun 17 2021 at 12:59)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> it seems to work indeed, thank you so much!</p>



<a name="243012901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012901">(Jun 17 2021 at 12:59)</a>:</h4>
<p>No problem but I think you've actually helped locate a problem!</p>



<a name="243012972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243012972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243012972">(Jun 17 2021 at 13:00)</a>:</h4>
<p>Something isn't right with the Linaro release!</p>



<a name="243013021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013021">(Jun 17 2021 at 13:00)</a>:</h4>
<p>If you answer the Qs above I can help clean up the scripts too. Just let me know.</p>



<a name="243013159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013159">(Jun 17 2021 at 13:01)</a>:</h4>
<p>so, about the script</p>



<a name="243013297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013297">(Jun 17 2021 at 13:02)</a>:</h4>
<p>we're not actually running that exact same script anywhere: the script just reproduces the commands Packer runs to be able to test the commands without Packer's overhead</p>



<a name="243013352"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013352" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013352">(Jun 17 2021 at 13:02)</a>:</h4>
<p>it was easier to play around changing a bash script rather than changing packer <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span></p>



<a name="243013360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013360">(Jun 17 2021 at 13:02)</a>:</h4>
<p>Ok Got it! No problem then.</p>



<a name="243013364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013364">(Jun 17 2021 at 13:02)</a>:</h4>
<p>:)</p>



<a name="243013411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013411">(Jun 17 2021 at 13:02)</a>:</h4>
<p>Let me know if things work out!</p>



<a name="243013436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243013436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243013436">(Jun 17 2021 at 13:03)</a>:</h4>
<p>For the migration, I mean.</p>



<a name="243015567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243015567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243015567">(Jun 17 2021 at 13:19)</a>:</h4>
<p>now I'm curious whether <a href="https://github.com/rust-lang/gha-self-hosted/issues/10">https://github.com/rust-lang/gha-self-hosted/issues/10</a> and <a href="https://github.com/rust-lang/gha-self-hosted/issues/8">https://github.com/rust-lang/gha-self-hosted/issues/8</a> were also caused by the UEFI blob</p>



<a name="243021225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243021225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243021225">(Jun 17 2021 at 13:58)</a>:</h4>
<p>ok the new blob fixed the graceful shutdown not working!</p>



<a name="243021281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243021281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243021281">(Jun 17 2021 at 13:58)</a>:</h4>
<p>seems not to have fixed ejecting the CD-ROM, but oh well that's pretty minor</p>



<a name="243021328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243021328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243021328">(Jun 17 2021 at 13:59)</a>:</h4>
<p>ok will do the switchover tomorrow</p>



<a name="243021497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243021497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243021497">(Jun 17 2021 at 14:00)</a>:</h4>
<p>thank you again <span class="user-mention" data-user-id="275830">@Robin Randhawa</span>! your help here was invaluable <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="243022931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243022931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243022931">(Jun 17 2021 at 14:09)</a>:</h4>
<p>No problem! <span aria-label="grinning face with smiling eyes" class="emoji emoji-1f601" role="img" title="grinning face with smiling eyes">:grinning_face_with_smiling_eyes:</span></p>



<a name="243026026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243026026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243026026">(Jun 17 2021 at 14:30)</a>:</h4>
<p>Can you provide me with the qemu invocation (or equivalent) for the CD-ROM eject problem ?</p>



<a name="243028115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243028115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243028115">(Jun 17 2021 at 14:43)</a>:</h4>
<p>Also FYI (and I learn something new everyday). The linaro QEMU UEFI binary releases URL points to abandonware... The expectation is that folk now use the distro specific packages. This has clearly not been communicated well!</p>



<a name="243028439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243028439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243028439">(Jun 17 2021 at 14:45)</a>:</h4>
<p><span class="user-mention silent" data-user-id="275830">Robin Randhawa</span> <a href="#narrow/stream/242791-t-infra/topic/new.20aarch64.20ci.20server/near/243028115">said</a>:</p>
<blockquote>
<p>Also FYI (and I learn something new everyday). The linaro QEMU UEFI binary releases URL points to abandonware... The expectation is that folk now use the distro specific packages. This has clearly not been communicated well!</p>
</blockquote>
<p>TIL!</p>



<a name="243141911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243141911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243141911">(Jun 18 2021 at 10:24)</a>:</h4>
<p>reprovisioning the new aarch64 server with a fresh ubuntu 20.04 installation</p>



<a name="243142457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243142457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243142457">(Jun 18 2021 at 10:30)</a>:</h4>
<p>as I'm reprovisioning everything anyway, regenerated the github token for gha-self-hosted (cc <span class="user-mention" data-user-id="116122">@simulacrum</span>)</p>



<a name="243143762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243143762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243143762">(Jun 18 2021 at 10:47)</a>:</h4>
<p>deploying the new server</p>



<a name="243145340"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243145340" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243145340">(Jun 18 2021 at 11:05)</a>:</h4>
<p>reopened the tree</p>



<a name="243145658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243145658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243145658">(Jun 18 2021 at 11:08)</a>:</h4>
<p>switched <a href="http://ci-arm-1.infra.rust-lang.org">ci-arm-1.infra.rust-lang.org</a> to point to the new instance</p>



<a name="243145716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243145716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243145716">(Jun 18 2021 at 11:09)</a>:</h4>
<p>and the new agent is working <span aria-label="tada" class="emoji emoji-1f389" role="img" title="tada">:tada:</span></p>



<a name="243145853"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243145853" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243145853">(Jun 18 2021 at 11:10)</a>:</h4>
<p><code>poweroff</code>'d the old server</p>



<a name="243145897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243145897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243145897">(Jun 18 2021 at 11:11)</a>:</h4>
<p>if CI builds work fine we can then deprovision the old machine</p>



<a name="243149586"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243149586" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243149586">(Jun 18 2021 at 11:51)</a>:</h4>
<p>whoa</p>



<a name="243149658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243149658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243149658">(Jun 18 2021 at 11:52)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> <span class="user-mention" data-user-id="275830">@Robin Randhawa</span> the <code>aarch64-gnu</code> went from 1hr45min to 45min!!!</p>



<a name="243149676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243149676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243149676">(Jun 18 2021 at 11:52)</a>:</h4>
<p>without giving more cores to the VM</p>



<a name="243152005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243152005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243152005">(Jun 18 2021 at 12:18)</a>:</h4>
<p>Sweet!!! <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span></p>



<a name="243243440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243243440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243243440">(Jun 19 2021 at 10:46)</a>:</h4>
<p><span class="user-mention" data-user-id="275830">@Robin Randhawa</span> thanks so much for the help!</p>
<p><span class="user-mention silent" data-user-id="275830">Robin Randhawa</span> <a href="#narrow/stream/242791-t-infra/topic/new.20aarch64.20ci.20server/near/243012185">said</a>:</p>
<blockquote>
<p>The biggest issue is that the UEFI binary is too old and buggy. The host system's qemu installation on Debian derived distros supplies a verified version in the qemu-efi-aarch64 package.</p>
</blockquote>
<p>if you get a minute, I'd love to know how you narrowed it down to this. did you enable assorted logging and the problem became relatively clear?</p>



<a name="243339716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243339716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243339716">(Jun 21 2021 at 04:51)</a>:</h4>
<p>I'd love to hear about that as well. Doing a lot of debugging on opaque EFI issues lately.</p>



<a name="243484955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243484955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243484955">(Jun 22 2021 at 08:20)</a>:</h4>
<p>deleted the old CI machine</p>



<a name="243553182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553182">(Jun 22 2021 at 17:46)</a>:</h4>
<p>Hi <span class="user-mention" data-user-id="117568">@Aidan Hobson Sayers</span> , <span class="user-mention" data-user-id="239881">@Josh Triplett</span> : Sure. So in an attempt to boil things down to a simple reproducer I found that this invocation reliably got me all the way to Linux:</p>
<p>qemu-system-aarch64 -m 2G -M virt -cpu max -bios QEMU_EFI.fd -drive if=none,file=rootfs.qcow2,id=hd0,cache=writeback,discard=unmap -device virtio-blk-device,drive=hd0 -nographic</p>



<a name="243553275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553275">(Jun 22 2021 at 17:46)</a>:</h4>
<p>On the other hand, this one reliably produced the issues that <span class="user-mention" data-user-id="121055">@Pietro Albini</span> was calling out (1 CPU stuck at 100% and no output):</p>
<p>qemu-system-aarch64 -m 2G -M virt -cpu max -machine gic-version=3 -bios QEMU_EFI.fd -drive if=none,file=rootfs.qcow2,id=hd0 -device virtio-blk-device,drive=hd0 -nographic</p>



<a name="243553458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553458">(Jun 22 2021 at 17:48)</a>:</h4>
<p>So I had a hunch that the problem is to with the GICv3 emulation for sure.</p>



<a name="243553567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553567">(Jun 22 2021 at 17:49)</a>:</h4>
<p>Also: both of those invocations were non KVM related so this was more fundamental.</p>



<a name="243553614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553614">(Jun 22 2021 at 17:49)</a>:</h4>
<p>Finally, what caught my eye was the URL used to fetch the EFI binary for qemu. That pointed to a binary built in 2018 (!).</p>



<a name="243553854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243553854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243553854">(Jun 22 2021 at 17:51)</a>:</h4>
<p>So rather than any funky spelunking all I decided to do was try out a new EFI binary. That worked without and with the KVM emulation.</p>



<a name="243554627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243554627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Robin Randhawa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243554627">(Jun 22 2021 at 17:56)</a>:</h4>
<p>What I would have done if I hadn't hit that hunch is a combination of '-d in_asm' and '-d exec' and symbolic info to find out where things were spinning.</p>



<a name="243893999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243893999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243893999">(Jun 25 2021 at 09:37)</a>:</h4>
<p>Thanks <span class="user-mention" data-user-id="275830">@Robin Randhawa</span>, that's very helpful!</p>



<a name="243902733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243902733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243902733">(Jun 25 2021 at 11:24)</a>:</h4>
<p>I wonder if some of that information is useful to put on our self hosted runner forge page for the future</p>



<a name="243902757"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243902757" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243902757">(Jun 25 2021 at 11:24)</a>:</h4>
<p><a href="https://forge.rust-lang.org/infra/docs/gha-self-hosted.html">https://forge.rust-lang.org/infra/docs/gha-self-hosted.html</a></p>



<a name="243902770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/new%20aarch64%20ci%20server/near/243902770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/new.20aarch64.20ci.20server.html#243902770">(Jun 25 2021 at 11:24)</a>:</h4>
<p><span class="user-mention" data-user-id="117568">@Aidan Hobson Sayers</span> do you think you might have some time to do so?</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>